Terminal device, method for acquiring drawing target, and computer-readable recording medium

ABSTRACT

A terminal device, comprising: a capturing unit capturing a subject; a light emitting unit; a display unit, a processor, and a storage unit containing a program that causes the processor to execute a process of acquiring a drawing target drawn by a user from a captured image of the subject including the drawing target and causing the drawing target to be displayed on a screen of the display unit, the program causes the processor to execute; controlling the light emitting unit to emit the light when the capturing unit captures the subject according to the user&#39;s operation, acquiring the captured image that is obtained by capturing the subject irradiated with the light emitted from the light emitting unit, performing image correction corresponding to a brightness distribution of the light emitted from the light emitting unit on the captured image, and extracting the drawing target from the captured image acquired.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to and incorporates by referencethe entire contents of Japanese Patent Application No. 2014-102745 filedin Japan on May 16, 2014.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a terminal device, a method foracquiring drawing target, and computer-readable recording mediumcontaining a computer program.

2. Description of the Related Art

In the past, a technique of converting an image obtained by a userdrawing a desired character on a sheet or the like into digital data,combining the digital data with another image data, and displayingcombined data has been known. Japanese Laid-open Patent Publication No.2009-131596 discloses a system in which it is possible to enjoy a racein a pseudo manner using characters drawn by respective users.

For example, a scanner device scans an original obtained by the userdrawing a character, and a computer connected to the scanner deviceextracts a character region from original image data obtained byscanning the original. The computer applies the image data in which thecharacter region is extracted to a coordinate space by a computer.

For example, the computer binarizes original image data throughthreshold determination based on brightness values of pixels, anddetermines a drawing region and a non-drawing region on the binarizedoriginal image data in units of pixels. Then, the computer uses theregion determined to be the drawing region as the character region.

Meanwhile, in the recent years, with the spread of multi-function mobileterminals such as smartphones or tablet-type computers, the performanceof a digital camera mounted in the multi-function mobile terminal hasbeen improved as well. If original image data is acquired by capturingan image of an original obtained by the user drawing a character throughthe digital camera mounted in the multi-function mobile terminal, theprocess of extracting the character region from the original image canbe implemented without using the scanner device.

Meanwhile, original image data obtained by capturing an original imagethrough a digital camera may be low in overall brightness or may havenon-uniform brightness within an original plane due to a shadow or thelike. In this case, it may be difficult to extract the drawing regionproperly although the binarization process is performed on the capturedoriginal image data.

In order to solve low brightness or non-uniform brightness in a capturedimage of an original image, a method of turning on a light installed inthe multi-function mobile terminal is considered. When the light isturned on, influence of ambient light is reduced, and the occurrence ofa shadow is suppressed. Further, as the light is turned on, lowbrightness can be expected to be solved. However, since the lightmounted in the multi-function mobile terminal is commonly a point lightsource, an image captured in a state in which the light is turned on isbright in front of the light, that is, near a center of an original butdark in a peripheral portion, and thus an image is likely to havenon-uniform brightness. In this case, there is a problem in that it maybe difficult to properly extract the drawing region from the capturedimage data.

In view of the above-mentioned conventional problems, there is a need tomake it possible to easily extract a drawing region from a capturedimage obtained by capturing an original.

SUMMARY OF THE INVENTION

It is an object of the present invention to at least partially solve theproblems in the conventional technology.

According to the present invention, there is provided a terminal device,comprising: a capturing unit that captures a subject; a light emittingunit that emits light; a display unit, a processor, and a storage unitthat contain a program that causes the processor to execute a process ofacquiring a drawing target drawn by a user from a captured image of thesubject including the drawing target and causing the drawing target tobe displayed on a screen of the display unit, the program causes theprocessor to execute; controlling the light emitting unit to emit thelight when the capturing unit captures the subject according to theuser's operation, acquiring the captured image that is obtained bycapturing the subject irradiated with the light emitted from the lightemitting unit, performing image correction corresponding to a brightnessdistribution of the light emitted from the light emitting unit on thecaptured image, and extracting the drawing target from the capturedimage acquired.

The present invention also provides a method for acquiring drawingtarget performed by a terminal device that includes a capturing unitthat captures a subject, a light emitting unit that emits light, adisplay unit, a processor, and a storage unit that contain a programthat causes the processor to execute a process of acquiring a drawingtarget drawn by a user from a captured image of the subject includingthe drawing target and causing the drawing target to be displayed on ascreen of the display unit, the method comprising; controlling the lightemitting unit to emit the light when the capturing unit captures thesubject according to the user's operation, acquiring the captured imagethat is obtained by capturing the subject irradiated with the lightemitted from the light emitting unit, performing image correctioncorresponding to a brightness distribution of the light emitted from thelight emitting unit on the captured image, and extracting the drawingtarget from the captured image acquired.

The present invention also provides a non-transitory computer-readablerecording medium that contains a computer program that is installed in aterminal device that includes a capturing unit that captures a subject,a light emitting unit that emits light, a display unit, a processor, anda storage unit that contain a program that causes the processor toexecute a process of acquiring a drawing target drawn by a user from acaptured image of the subject including the drawing target and causingthe drawing target to be displayed on a screen of the display unit, theprogram causes the processor to execute; controlling the light emittingunit to emit the light when the capturing unit captures the subjectaccording to the user's operation, acquiring the captured image that isobtained by capturing the subject irradiated with the light emitted fromthe light emitting unit, performing image correction corresponding to abrightness distribution of the light emitted from the light emittingunit on the captured image, and extracting the drawing target from thecaptured image acquired.

The above and other objects, features, advantages and technical andindustrial significance of this invention will be better understood byreading the following detailed description of presently preferredembodiments of the invention, when considered in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1C are diagrams for conceptually describing a firstembodiment;

FIG. 2 is a block diagram illustrating an exemplary configuration of aninformation processing device according to the first embodiment;

FIG. 3 is an exemplary flowchart illustrating an image correctionprocess according to the first embodiment;

FIG. 4 is a diagram illustrating an exemplary characteristic of astandard image according to the first embodiment;

FIG. 5 is a block diagram more specifically illustrating an exemplaryconfiguration of an information processing device according to the firstembodiment;

FIG. 6 is a diagram illustrating an exemplary sheet on which a frameimage is formed according to a first modified example of the firstembodiment;

FIG. 7 is a block diagram illustrating an exemplary configuration of aninformation processing device according to the first modified example ofthe first embodiment;

FIG. 8 is an exemplary flowchart illustrating an image correctionprocess according to the first modified example of the first embodiment;

FIG. 9 is a block diagram illustrating an exemplary configuration of aninformation processing device according to a second modified example ofthe first embodiment;

FIG. 10 is a diagram illustrating a state in which a frame guide imageis displayed on a finder screen according to the second modified exampleof the first embodiment;

FIG. 11 is an exemplary flowchart illustrating an image correctionprocess according to the second modified example of the firstembodiment;

FIG. 12 is a diagram illustrating a state in which an image of a sheetis displayed on a finder screen on which a frame guide image isdisplayed according to the second modified example of the firstembodiment;

FIG. 13 is a block diagram illustrating an exemplary configuration of aninformation processing device according to a third modified example ofthe first embodiment;

FIG. 14 is an exemplary flowchart illustrating an image correctionprocess according to the third modified example of the first embodiment;

FIGS. 15A to 15D are diagrams for describing a change in a brightnessdistribution on a subject according to a distance between a light andthe subject;

FIG. 16 is a block diagram illustrating an exemplary configuration of aninformation processing device according to a second embodiment;

FIG. 17 is an exemplary flowchart illustrating an image correctionprocess according to the second embodiment;

FIG. 18 is a block diagram illustrating an exemplary configuration of aninformation processing device that can be applied to the embodiments andthe modified examples;

FIG. 19 is a block diagram illustrating an exemplary configuration of adisplay system according to a third embodiment;

FIG. 20 is a diagram schematically illustrating an image data spacehaving a 3D coordinate system according to the third embodiment;

FIG. 21 is a block diagram illustrating an exemplary configuration of aPC according to the third embodiment.

FIG. 22 is an exemplary functional block diagram for describing afunction of the PC according to the third embodiment;

FIG. 23 is an exemplary flowchart illustrating an overall flow of adisplay control process performed by the display system according to thethird embodiment; and

FIG. 24 is a diagram illustrating an exemplary sheet for performingdrawing by hand according to the third embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of a terminal device, a method for acquiring adrawing target, and a computer-readable recording medium containing acomputer program will be described in detail with reference to theappended drawings.

First Embodiment

A first embodiment will be described. For example, a first embodimentrelates to image processing that is performed on a captured image inorder to easily perform a process of extracting an image (referred to asa “user image”) that is drawn by the user and included in a capturedimage obtained by capturing a sheet having an image drawn by the userthrough an imaging device. The user image means a drawing target thatdrawn by a user. Thus, the drawing target includes illustrations andpictures where not only animals, plants, game characters, but alsowriting characters are freely drawn and written by a user.

Overview of First Embodiment

The first embodiment will be conceptually described with reference toFIGS. 1A to 1G. For example, the user draws an arbitrary user image 301on a sheet 300 as illustrated in FIG. 1A. In order to convert the userimage 301 into image data, the sheet 300 including the user image 301 iscaptured using a digital camera. Image processing is performed on acaptured image obtained by capturing the sheet 300, and a region of theuser image 301 is extracted. For example, the user image 301 indicates aline drawn by the user, and the region of the user image 301 indicates aregion including the entire user image 301.

At this time, for example, a process of performing thresholddetermination on pixel values (brightness values) of pixels of thecaptured image, binarizing the captured image, and deciding the regionof the user image 301 based on the binarized pixels can be applied asthe image processing. For example, binarization of setting a pixelhaving a pixel value equal to or larger than a threshold to a value “1”and setting a pixel having a pixel value less than a threshold to avalue “0” is performed, and pixels of the value “0” are decided to bepixels by the user image 301. Further, the region of the user image 301extracted based on the pixels of the value “0.”

When the binarization process is performed on the captured image, ifthere is a shadow on the sheet 300, a shadow portion is extracted as thevalue “0” through the threshold determination, and thus it is likely tobe difficult to extract the user image 301 properly. Further, this isthe same even when lighting is insufficient, and overall brightness of acaptured image is low. Thus, it is desirable to irradiate a subject withlight or the like and suppress influence of a shadow or low brightnessat the time of capturing.

Here, a camera (hereinafter, referred to as an “internal camera”)embedded in a multi-function mobile terminal (user's terminal device)such as a smartphone or a tablet-type computer is assumed to be used asa digital camera. A light embedded in the multi-function mobile terminalis commonly regarded to be a point light source. Thus, when the sheet300 is irradiated with the light (hereinafter, referred to as an“internal light”) embedded in the multi-function mobile terminal, abrightness distribution in which brightness is highest in front of thelight, that is, near the center of the sheet 300, and brightnessgradually decreases from the center to the periphery thereof is formed.

FIG. 1B illustrates an exemplary captured image obtained by irradiatingthe sheet 300 with the light embedded in the multi-function mobileterminal and capturing the sheet 300. Referring to FIG. 1B, a capturedimage 310 has a brightness distribution in which brightness graduallydecreases toward the periphery as indicated by a gradation 311,corresponding to a brightness distribution of the sheet 300. If thebinarization process is performed on the captured image 310 having thisbrightness distribution, particularly, a peripheral region having lowbrightness is likely to be extracted as the user image together with auser image 301′. In this case, it is difficult to extract a region ofthe user image 301′ properly from the captured image 310.

In this regard, in the first embodiment, an image having a brightnessdistribution when a subject is irradiated with light is prepared as astandard image in advance, and a captured image captured by irradiatingwith light is corrected using the standard image.

FIG. 1C illustrates an exemplary standard image. Referring to FIG. 1C, astandard image 320 has a brightness distribution in which brightness hasa highest value near the center, and brightness gradually decreases fromthe center toward the periphery as indicated by a gradation 321. Forexample, a brightness change in the standard image is identical to abrightness change in the captured image. In other words, the standardimage 320 is an image having a brightness distribution of a plane by alight at a distance given as a distance of the sheet 300 from a camerawhen the captured image 310 is captured.

An operation indicated by the following Formulas (1) to (3) is performedin units of pixels between the captured image 310 and the standard image320.

R(i)=Ra(i)+(Y _(MAX) −Rb(i))  (1)

G(i)=Ga(i)+(Y _(MAX) −Gb(i))  (2)

B(i)=Ba(i)+(Y _(MAX) −Bb(i))  (3)

In Formula (1), a variable i is an index identifying a pixel, andvariables Ra(i), Ga(i), and Ba(i) indicate pixel values of R (red), G(green), and B (blue) of the pixel i in the captured image 310.Variables Rb(i), Gb(i), and Bb(i) indicate pixel values of R, G and B ofthe pixel i in the standard image 320. Y_(MAX) is a maximum value of apixel value common to colors of R, G, and B. When each of the pixels ofR, G, and B has a bit depth of 8 bits, Y_(MAX) is 255.

The pixel values R(i), G(i) and B(i) obtained as a result of theoperation of Formulas (1) to (3) are pixel values of a corrected imageobtained by correcting the captured image 310 using the standard image320. In the corrected image, a component of the standard image 320 isdeducted from the captured image 310, a pixel of a portion common to thecaptured image 310 and the standard image 320 has the pixel valueY_(MAX), and a pixel value of the captured image 310 are held for apixel of a portion of the captured image 310 different from the standardimage 320. Thus, the user image 301′ is held, and the pixel valueY_(MAX) is used for the other portion. In addition, the pixel valuesR(i), G(i) and B(i) is an example of a corrective value indicatingdegree of the correction to the captured image. Thus, parameterindicating the corrective value is not limited to the pixel value.

As the captured image 310 is corrected using the standard image 320 asdescribed above, pixel values of portions of the captured image 310other than the user image 301′ become uniform, and thus the accuracy ofthreshold determination for binarization is improved. Thus, it ispossible to extract the region of the user image 301′ from the capturedimage 310 easily and accurately.

More Specific Example of First Embodiment

Next, a more specific example of the first embodiment will be described.FIG. 2 illustrates an exemplary configuration of an informationprocessing device 400 according to the first embodiment. The informationprocessing device 400 includes a correcting unit 401, an operating unit402, a light 403, and a camera module 404. The correcting unit 401includes a control unit 410, a correction processing unit 411, and astandard image output unit 412. The information processing device 400functions as a multi-function mobile terminal, and the correcting unit401, the operating unit 402, the light 403, and the camera module 404are configured to be portable as a single body.

In the information processing device 400, the operating unit 402includes various kinds of operators receiving the user's operation. Aso-called touch panel in which a display device is integrated with aninput device that outputs a control signal according to a contactposition may be used as the operating unit 402. The light 403 is a lightemitting unit that emits light according to control of the control unit410. For example, the light 403 is a non-directional light sourceserving as a point light source. The light 403 is not limited to a pointlight source, and a light source having other characteristics may beused.

The camera module 404 includes a capturing element such as a ChargeCoupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS)imager, a drive circuit that drives a capturing element, and a capturingsignal processing unit that performs certain signal processing on acapturing signal output from the capturing element, and outputs acaptured image serving as digital data. For example, the camera module404 outputs the captured image with a certain frame period.

The correcting unit 401 performs the correction using the standard imageon the captured image output from the camera module 404, and outputs anoutput image. In the correcting unit 401, the control unit 410 controlsan operation of the information processing device 400 in general.According to control of the control unit 410, the correction processingunit 411 performs the operation indicated by Formulas (1) to (3) on thecaptured image output from the camera module 404 using the standardimage, and outputs an output image in which the captured image iscorrected. The standard image output unit 412 outputs the standard imageaccording to an instruction given from the correction processing unit411.

For example, the correcting unit 401 includes a Central Processing Unit(CPU), a Read Only Memory (ROM), and a Random Access Memory (RAM),operates according to an information processing program previouslystored in the ROM using the RAM as a work memory, and executes acorrection process on the captured image according to the firstembodiment. Each of the control unit 410, the correction processing unit411, and the standard image output unit 412 may be configured as amodule of the information processing program. The information processingprogram is an application program operating on the informationprocessing device 400, and is appropriately referred to as “app.”

Each unit included in the correcting unit 401 is not limited to anexample operating as a module of an information processing program. Forexample, some or all of the control unit 410, the correction processingunit 411, and the standard image output unit 412 included in thecorrecting unit 401 may be configured by hardware collaborating witheach other.

FIG. 3 is an exemplary flowchart illustrating an image correctionprocess according to the first embodiment. For example, when an appaccording to the first embodiment is activated according to the user'soperation on the operating unit 402 in step S100, in step S101, thecontrol unit 410 turns on the light 403 to emit light. The light 403 maybe turned on according to the user's operation on the operating unit402. In step S102, the control unit 410 is on standby for a capturinginstruction. For example, when a capturing instruction operation isdetermined to have been performed on the operating unit 402, the controlunit 410 causes the process to proceed to step S103. On the other hand,when no capturing instruction operation is determined to have beenperformed on the operating unit 402, the control unit 410 causes theprocess to return to step S102.

Further, when the sheet 300 on which the user image 301 is drawn iscaptured, it is desirable to keep a capturing plane in the camera module404 and the sheet 300 to be as parallel as possible. Further, it isdesirable to adjust a distance between the sheet 300 and the informationprocessing device 400 or an angle of view by the camera module 404 sothat the sheet 300 is captured to be as large as possible in aneffective capturing region of the capturing element in the camera module404.

In step S103, the correction processing unit 411 acquires the capturedimage 310 output from the camera module 404. In other words, thecorrection processing unit 411 functions as a captured image acquiringunit that acquires the captured image 310 output from the camera module404. The correction processing unit 411 stores the acquired capturedimage 310, for example, in a frame memory (not illustrated). Afteracquiring the captured image 310 through the correction processing unit411 in step S103, the control unit 410 may turn off the light 403.

In step S104, the correction processing unit 411 acquires the standardimage 320 from the standard image output unit 412. For example, thecorrection processing unit 411 requests the standard image output unit412 to send the standard image 320, and the standard image output unit412 outputs the standard image 320 to the correction processing unit 411in response to the request.

Here, the standard image output unit 412 may read and output thestandard image 320 that is generated and stored in the ROM or the likein advance, or may calculate and output data of the standard image 320according to a certain calculation formula or the like in units ofpixels. FIG. 4 illustrates an exemplary characteristic of the standardimage 320 according to the first embodiment. Referring to FIG. 4, avertical axis indicates brightness, that is, brightness Y, and ahorizontal axis indicates a distance d from an image center serving asthe front of the light 403. As illustrated in FIG. 4, in the standardimage 320, the brightness Y decreases according to the distance d fromthe image center. The standard image output unit 412 may include afunction or a table indicating a relation between the brightness Y andthe distance d and obtain pixel values of pixels of the standard image320 based on the function or the table.

In step 3105, the correction processing unit 411 corrects the capturedimage 310 acquired in step S103 using the standard image 320 acquired instep S104. At this time, the correction processing unit 411 performs theoperation according to Formulas (1) to (3) for each of the pixels of thestandard image 320 and the captured image 310, and corrects the capturedimage 310.

For example, the correction processing unit 411 causes the standardimage 320 acquired in step S104 to be stored in the RAM or the like. Thecorrection processing unit 411 reads the standard image 320 from the RAMin units of pixels, and reads the pixel of the captured image 310 whoseposition corresponds to the read pixel from the frame memory in whichthe captured image 310 is stored in step S103. Then, the operationindicated by Formulas (1) to (3) is performed using the pixel of thestandard image 320 and the pixel of the captured image 310 that are readfrom the RAM and the frame memory.

The present invention is not limited to this example, and the correctionprocessing unit 411 may receive the pixel of the standard image 320 fromthe standard image output unit 412 in units of pixels in step S104 andthen perform the correction process.

In step S106, the correction processing unit 411 acquires the correctedimage obtained by correcting the captured image 310 in step S105. Thecorrection processing unit 411 outputs the acquired corrected image tothe outside of the correcting unit 401 as an output image. For example,the output image is supplied to an image extracting unit, thebinarization process is performed to decide the pixels of the user image301′, and the region of the user image 301′ is extracted based on thedecided pixels.

The standard image 320 has been described to be given in advance, butthe present invention is not limited to this example. For example, thestandard image 320 may be generated before the sheet 300 on which theuser image 301 is drawn is captured. For example, in the informationprocessing device 400, actually, the light 403 is caused to emit light,and an angle of view by the camera module 404 is caused to match anangle of view at which the sheet 300 on which the user image 301 isdrawn is captured. In this state, the user image 301 is not drawn, and anew sheet having the same ground color as the sheet 300 is captured. Thecaptured image 310 may be used as the standard image 320. In this case,even when the sheet 300 is not white, the correction process can beperformed.

As described above, in the first embodiment, the captured image 310obtained by capturing the sheet 300 in the state in which the light 403is turned on is corrected using the standard image 320 having thebrightness distribution of the plane by irradiation of the light 403 atthe distance given as the distance of the sheet 300 from the camera (thecapturing element of the camera module 404) when the captured image 310is captured. Thus, the corrected image obtained by correcting thecaptured image 310 is uniform in the brightness distribution byirradiation of the light 403, and thus the user image 301′ is easilyextracted.

FIG. 5 more specifically illustrates an exemplary configuration of aninformation processing device according to the first embodiment. In FIG.5, the same components as in FIG. 2 are denoted by the same referencenumerals, and a detailed description thereof is omitted. Referring toFIG. 5, an information processing device 400 a includes an operatingunit 402, a light 403, a control unit 430 a, a capturing unit 440, acapturing signal processing unit 441, frame memories 442 and 445, animage processing unit 443, a display unit 444, a standard image outputunit 446, and an operation unit 447.

Of these, the capturing unit 440 and the capturing signal processingunit 441 are included in the camera module 404 of FIG. 2. The controlunit 430 a corresponds to the control unit 410 of FIG. 2. Further, thecontrol unit 430 a has a function of the captured image acquiring unitin the correction processing unit 411 of FIG. 2. The frame memories 442and 445, the image processing unit 443, and the operation unit 447 areincluded in the correction processing unit 411. The standard imageoutput unit 44E corresponds to the standard image output unit 412, andoutputs the image data (standard image data) of the standard image. Thedisplay unit 444 includes a display device such as a Liquid CrystalDisplay (LCD) and a drive circuit that drives the display device.

A control signal according to the user's operation is input from theoperating unit 402 to the control unit 430 a. The control unit 430 acontrols lighting of the light 403. As will be described later, thecontrol unit 430 a controls a read/write timing of the frame memory 445and an operation of the operation unit 447.

The capturing unit 440 outputs an analog capturing signal with a frameperiod according to a clock (not illustrated). The analog capturingsignal output from the capturing unit 440 is input to the capturingsignal processing unit 441. The capturing signal processing unit 441performs certain signal processing such as noise reduction and gaincontrol on the input capturing signal. The capturing signal processingunit 441 performs A/D conversion on the analog capturing signal that hasbeen subjected to the certain signal processing to convert the analogcapturing signal into digital captured image data, and outputs thedigital captured image data. The captured image data output from thecapturing signal processing unit 441 is stored in the frame memory 442in units of frames.

The image processing unit 443 reads the captured image data from theframe memory 442 in units of frames, performs image processing such as agamma correction process and a white balance adjustment process on theread captured image data, and outputs resultant data. For example, theimage data that is obtained by performing the image processing on thecaptured image data and output from the image processing unit 443 issupplied to the display unit 444 and displayed on the display device.Thus, it is possible to cause the display unit 444 to function as afinder used to check the captured image captured by the capturing unit440.

The control unit 430 a causes captured image data of one frame stored inthe frame memory 442 to be stored in the frame memory 445 according to acapturing instruction on the operating unit 402, and acquires capturedimage data of a processing target (step 3103 of FIG. 3). The operationunit 447 performs the operation according to Formulas (1) to (3) betweenthe standard image output from the standard image output unit 446 andthe captured image data stored in the frame memory 445 according tocontrol of the control unit 430 a, so that corrected image data in whichthe captured image data is corrected is acquired. The operation unit 447outputs the corrected image data as an output image data.

First Modified Example of First Embodiment

Next, a first modified example of the first embodiment will bedescribed. In the first embodiment, the user adjusts the distance (anangle of view by the capturing unit 440) between the informationprocessing device 400 a and the sheet 300 so that the user image 301 isincluded in the captured image while checking the display of the displayunit 444, and performs the user's operation to give the capturinginstruction.

In the first modified example of the first embodiment, a frame imageindicating a region in which the user is allowed to draw the user image301 on the sheet 300 is formed in advance. Further, the informationprocessing device recognizes a frame image included in a captured image,and acquires the captured image automatically when the frame image isdetected from the captured image. The user can easily acquire thecaptured image appropriately including the user image 301 and acquirethe captured image without performing an operation to give the capturinginstruction to the information processing device.

FIG. 6 illustrates an exemplary sheet on which a frame image is formedaccording to the first modified example of the first embodiment.Referring to FIG. 6, a frame image 331 indicating a region 332 in whichthe user image 301 drawn is formed on a sheet 330 in advance. In theexample of FIG. 6, in addition to the rectangular frame image 331,markers 333 a, 333 b, and 333 c are formed at three positions among fourpositions of the frame image 331. An outer edge of the region 332 and adirection of the region 332 can be determined by detecting the markers333 a, 333 b, and 333 c at the three positions by image recognition. Atitle of the user image 301 drawn in the region 332 can be written in aregion 334.

FIG. 7 illustrates an exemplary configuration of an informationprocessing device according to the first modified example of the firstembodiment. In FIG. 7, the same components as in FIG. 5 are denoted bythe same reference numerals, and a detailed description thereof isomitted.

Referring to FIG. 7, an information processing device 400 b has aconfiguration in which a frame recognizing unit 450 a is added to theinformation processing device 400 a of FIG. 5. The frame recognizingunit 450 a recognizes the frame image 331 included in the image dataoutput from the image processing unit 443, and detects the frame image331.

For example, the frame recognizing unit 450 a is considered to storeimage pattern information corresponding to the frame image 331 inadvance and detect the frame image 331 by comparing the image patterninformation with the image data. The frame recognizing unit 450 a maydetect the frame image 331 based on the image pattern informationcorresponding to the markers 333 a to 333 c. Further, the framerecognizing unit 450 a determines whether or not the frame image 331 isrecognized at a certain position within the image data. Thus, the frameimage 331 of a certain size included in the captured image can berecognized.

FIG. 8 is an exemplary flowchart illustrating an image correctionprocess according to the first modified example of the first embodiment.For example, when an app according to the first modified example of thefirst embodiment is activated according to the user's operation on theoperating unit 402 in step S110, in step Sill, a control unit 430 bturns on the light 403 to emit light.

In step S112, the frame recognizing unit 450 a determines whether or notthe frame image 331 is recognized from the image data output from theimage processing unit 443. When the frame image 331 is determined to benot recognized, the frame recognizing unit 450 a causes the process toreturn to step S112. On the other hand, when the frame image 331 isdetermined to be recognized in the image data, the frame recognizingunit 450 a causes the process to proceed to step S113.

In step S113, the frame recognizing unit 450 a determines whether or notthe frame image 331 recognized in step S112 is at a certain positionwithin the image data. When the frame image 331 is determined to be notat the certain position, the frame recognizing unit 450 a causes theprocess to return to step S112. On the other hand, when the frame image331 is determined to be not at the certain position within the imagedata, the frame recognizing unit 450 a causes the process to proceed tostep S114.

In step S114, the control unit 430 b reads the captured image data ofone frame from the frame memory 442, stores the captured image data ofone frame in the frame memory 445, and acquires the captured image dataof the processing target. In step S115, the control unit 430 b acquiresthe standard image data output from the standard image output unit 446.For example, the control unit 430 b controls the operation unit 447 suchthat the standard image data output from the standard image output unit446 is input to the operation unit 447.

In step S116, the operation unit 447 corrects the captured image datastored in the frame memory 445 in step S114 using the standard imagedata input in step S115 according to Formulas (1) to (3). The operationunit 447 acquires the corrected image data that is corrected by thecorrection process (step S117). The operation unit 447 outputs theacquired corrected image data to, for example, an image extracting unit(not illustrated).

Second Modified Example of First Embodiment

Next, a second modified example of the first embodiment will bedescribed. In the second modified example, in addition to the framerecognition function according to the first modified example, a frameguide image used to position the frame image 331 is displayed on afinder screen used to check a capturing state displayed on the displayunit 444. In other word, the finder screen allows the user to confirmvisually the drawing target to be captured. Further, it is determinedwhether or not the frame image 331 is displayed within a certain rangeon the frame guide image in the finder screen.

FIG. 9 illustrates an exemplary configuration of an informationprocessing device according to the second modified example of the firstembodiment. In FIG. 9, the same components as in FIG. 5 are denoted bythe same reference numerals, and a detailed description thereof isomitted. Referring to FIG. 9, an information processing device 400 c hasa configuration in which a frame recognizing unit 450 b, a framegenerating unit 451, and a synthesizing unit 452 are added to theinformation processing device 400 a of FIG. 5.

The frame generating unit 451 generates a frame guide image to bedisplayed on a finder screen. The synthesizing unit 452 synthesizes theframe guide image generated by the frame generating unit 451 with theimage data output from the image processing unit 443 according tocontrol of a control unit 430 c, and generates image data with a guidein which the frame guide image is arranged at a certain position in animage based on the image data. FIG. 10 illustrates a state in which aframe guide image 341 is displayed in a finder screen 340 according tothe second modified example of the first embodiment. The framegenerating unit 451 transfer information indicating coordinates of theframe guide image 341 within the image data to the frame recognizingunit 450 b.

Similarly to the frame recognizing unit 450 a, the frame recognizingunit 450 b recognizes the frame image 331 included in the image datathat is output from the image processing unit 443 and supplied throughthe synthesizing unit 452, and detects the frame image 331. Further, theframe recognizing unit 450 b determines whether or not the frame image331 detected from the image data falls within a certain range on theframe guide image 341 generated by the frame generating unit 451 basedon the information indicating the coordinates of the frame guide image341 transferred from the frame generating unit 451.

FIG. 11 is an exemplary flowchart illustrating an image correctionprocess according to the second modified example of the firstembodiment. For example, when an app according to the second modifiedexample of the first embodiment is activated according to the user'soperation on the operating unit 402 in step S120, in step S121, thecontrol unit 430 c turns on the light 403 to emit light.

In step S122, the frame generating unit 451 generates the frame guideimage 341, supplies the frame guide image 341 to the synthesizing unit452, and causes the frame guide image 341 to be displayed within thefinder screen 340. At this time, the frame generating unit 451 transfersthe information indicating the coordinates of the frame guide image 341in the finder screen 340, that is, information indicating thecoordinates within the image data output from the image processing unit443 to the frame recognizing unit 450 b.

In step S123, the frame recognizing unit 450 b determines whether or notthe frame image 331 is recognized from the image data output from theimage processing unit 443. When the frame image 331 is determined to benot recognized, the frame recognizing unit 450 b causes the process toreturn to step S123. On the other hand, when the frame image 331 isdetermined to be recognized in the image data, the frame recognizingunit 450 b causes the process to proceed to step S124.

In step S124, the frame recognizing unit 450 b determines whether or notthe frame image 331 included in the image data is displayed within acertain range on the frame guide image 341 based on the coordinateinformation of the frame guide image 341.

FIG. 12 illustrates a state in which an image of the sheet 330 isdisplayed on the finder screen 340 on which the frame guide image 341 isdisplayed according to the second modified example of the firstembodiment. In the example of FIG. 12, the entire frame image 331 formedon the sheet 330 is included in the frame guide image 341, and adistance between the frame image 331 and the frame guide image 341 fallswithin a certain range. Thus, the frame recognizing unit 450 bdetermines that the frame image 331 is displayed within the certainrange on the frame guide image 341.

Here, the distance between the frame image 331 and the frame guide image341 is a distance between corresponding sides of the frame image 331 andthe frame guide image 341. In step S124, the frame recognizing unit 450b performs distance determination of sides based on coordinates of theframe image 331 and the frame guide image 341 within the image data, anddetermines whether or not a relation between the frame image 331 and theframe guide image 341 is a state illustrated in FIG. 12. The framerecognizing unit 450 b may perform the distance determination bycomparing coordinates of four corners of the frame image 331 withcoordinates of four corners of the frame guide image 341 instead ofsides of the frame image 331 and the frame guide image 341.

In the example illustrated in FIG. 12, the frame image 331 is includedin the frame guide image 341, but the present invention is not limitedto this example. In other words, even when the frame guide image 341 isincluded within the frame image 331, the frame recognizing unit 450 bdetermines that the frame image 331 is displayed within the certainrange on the frame guide image 341 if the distance between the frameimage 331 and the frame guide image 341 falls within the certain range.

For example, the user adjust the distance between the informationprocessing device 400 c and the sheet 330 so that the frame image 331 isdisplayed within a certain range on the frame guide image 341 whilechecking the finder screen 340 displayed on the display unit 444.

When the frame image 331 is determined to be not displayed within thecertain range on the frame guide image 341 in step S124, the framerecognizing unit 450 b causes the process to return to step S123. On theother hand, when the frame image 331 is determined to be displayedwithin the certain range on the frame guide image 341, the framerecognizing unit 450 b causes the process to proceed to step S125.

In step S125, the control unit 430 c reads the captured image data ofone frame from the frame memory 442, and causes the captured image dataof one frame to be stored in the frame memory 445, and acquires thecaptured image data of the processing target. In step S126, the controlunit 430 c acquires the standard image data output from the standardimage output unit 446. For example, the control unit 430 c controls theoperation unit 447 such that the standard image data output from thestandard image output unit 446 is input to the operation unit 447.

In step S127, the operation unit 447 corrects the captured image datastored in the frame memory 445 in step S125 according to Formulas (1) to(3) using the standard image data input in step S126. The operation unit447 acquires the corrected image data that is corrected by thecorrection process (step S128). The operation unit 447 outputs theacquired corrected image data to, for example, an image extracting unit(not illustrated).

As described above, in the second modified example of the firstembodiment, since the frame guide image 341 is displayed within thefinder screen 340, the process of capturing the frame image 331 in thecaptured image with an appropriate size can be easily performed.Further, it is possible to acquire the captured image without performingan operation to give the capturing instruction to the informationprocessing device 400 c.

Third Modified Example According to First Embodiment

Next, a third modified example according to the first embodiment will bedescribed. When the sheet 330 is captured, if the capturing plane of thecapturing element of the capturing unit 440 is not parallel to the sheet330, trapezoidal distortion occurs in the captured image of the sheet330. More specifically, the frame image 331 that has to be capturedoriginally in a rectangular shape becomes an image deformed in atrapezoidal shape according to an angle of the capturing plane to thesheet 330 in the captured image. In this case, the brightnessdistribution of the captured image by light irradiation of the light 403is different from the brightness distribution of the standard image, andalthough the operation according to Formulas (1) to (3) is performed,pixel values are not uniform.

In this regard, in the third modified example according to the firstembodiment, an inclination of an information processing device isdetected, and when the inclination is equal to or larger than a certainvalue, the acquisition of the captured image is not performed. Here, theinclination is one specific example of shape conditions of theinformation processing device.

FIG. 13 illustrates an exemplary configuration of an informationprocessing device according to the third modified example according tothe first embodiment. In FIG. 13, the same components as in FIG. 9 aredenoted by the same reference numerals, and a detailed descriptionthereof is omitted. Referring to FIG. 13, an information processingdevice 400 d has a configuration in which an inclination sensor 460 isadded to the information processing device 400 c of FIG. 9.

For example, the inclination sensor 460 detects the inclination of theinformation processing device 400 d using a gyroscope. The inclinationsensor 460 is assumed to be able to detect at least an angle to ahorizontal plane as an inclination. The inclination sensor 460 outputsthe angle to the horizontal plane as a detection result to be suppliedto a control unit 430 d.

FIG. 14 is an exemplary flowchart illustrating an image correctionprocess according to the third modified example according to the firstembodiment. The sheet 330 that includes the user image 301 drawn thereonand serves as the capturing target of the information processing device400 d is assumed to be maintained to be as horizontal as possible. Forexample, when an app according to the third modified example accordingto the first embodiment is activated according to the user's operationon the operating unit 402 in step S140, in step S141, the control unit430 d turns on the light 403 to emit light.

In step S142, the frame generating unit 451 generates the frame guideimage 341, supplies the frame guide image 341 to the synthesizing unit452, causes the frame guide image 341 to be displayed on the finderscreen 340, and transfer the information indicating the coordinates ofthe frame guide image 341 in the finder screen 340 to the framerecognizing unit 450 b.

In step S143, the frame recognizing unit 450 b determines whether or notthe frame image 331 is recognized from the image data output from theimage processing unit 443. When the frame image 331 is determined to berecognized, the frame recognizing unit 450 b causes the process toreturn to step S143. On the other hand, when the frame image 331 isdetermined to be recognized in the image data, the frame recognizingunit 450 b causes the process to proceed to step S144.

In step S144, the frame recognizing unit 450 b determines whether or notthe frame image 331 included in the image data is displayed within acertain range on the frame guide image 341 based on the coordinateinformation of the frame guide image 341. When the frame image 331 isdetermined to be not displayed within the certain range on the frameguide image 341, the frame recognizing unit 450 b causes the process toreturn to step S143. On the other hand, when the frame image 331 isdetermined to be displayed within the certain range on the frame guideimage 341, the frame recognizing unit 450 b causes the process toproceed to step S145.

In step 3145, the control unit 430 d determines whether or not theinclination to the horizontal plane of the information processing device400 d is within a certain range based on the detection result suppliedfrom the inclination sensor 460. When the inclination is determined toexceed the certain range, the control unit 430 d causes the process toreturn to step S143. On the other hand, when the inclination isdetermined to be within the certain range, the control unit 430 d causesthe process to proceed to step S146.

In step S146, the control unit 430 d reads the captured image data ofone frame from the frame memory 442, causes the captured image data ofone frame to be stored in the frame memory 445, and acquires thecaptured image data of the processing target. In step S147, the controlunit 430 d acquires the standard image data output from the standardimage output unit 446. In step S148, the operation unit 447 corrects thecaptured image data stored in the frame memory 445 in step S146according to Formulas (1) to (3) using the standard image data input instep S147. The operation unit 447 acquires the corrected image data thatis corrected by the correction process (step S149). The operation unit447 outputs the acquired corrected image data to an image extractingunit (not illustrated).

As described above, in the third modified example according to the firstembodiment, the inclination of the information processing device isdetected, and when the inclination is equal to or larger than a certainvalue, the acquisition of the captured image is not performed. Thus, itis possible to prevent a situation in which trapezoidal distortionoccurs in the captured image since the information processing device 400d is inclined.

Second Embodiment

Next, a second embodiment will be described. In the first embodiment,when the sheet 300 on which the user image 301 is drawn is captured, theuser adjusts the position of the information processing device 400 sothat the distance between the information processing device 400 and thesheet 300 becomes a certain distance. This is the same in the modifiedexamples of the first embodiment as well as the first embodiment.

In other words, a tendency of a brightness distribution by lightirradiation of the light 403 on a subject changes according to thedistance between the light 403 and the subject as illustrated in FIGS.15A to 15D. In FIGS. 15A to 15D, FIG. 15A illustrates an example of abrightness distribution when the distance between the light 403 and thesubject is largest, and FIGS. 155, 15C, and 15D illustrate examples ofbrightness distributions when the distance between the light 403 and thesubject is sequentially decreased in the described order.

In this example, as the distance between the light 403 and the subjectincreases, the overall brightness of the subject decreases, and thebrightness of the peripheral portion further decreases. Conversely, asthe distance between the light 403 and the subject decreases, theoverall brightness of the subject increases, and the brightness of theperipheral portion increases as well.

Thus, a plurality of standard images corresponding to the brightnessdistributions according to each distance are prepared, the distance fromthe light 403 is measured using a distance sensor, and a standard imageaccording to the distance from the light 403 is selected from aplurality of standard images based on a measurement result. As thecaptured image is corrected using the selected standard image selectedaccording to the distance, it is possible to obtain an appropriatecorrected image by adjusting the position of the information processingdevice roughly.

FIG. 16 illustrates an exemplary configuration of an informationprocessing device according to the second embodiment. In FIG. 16, thesame components as in FIG. 5 are denoted by the same reference numerals,and a detailed description thereof is omitted. Referring to FIG. 16, aninformation processing device 400 e has a configuration in which adistance sensor 470 is added to the information processing device 400 aof FIG. 5. As the distance sensor 470, a distance sensor that measures adistance to a subject can be shared when the capturing is performed bythe capturing unit 440.

Referring to FIG. 16, a standard image output unit 480 includes aplurality of standard image data 481 a, 481 b, 481 c, and 481 dcorresponding to a plurality of distances #1, #2, #3, and #4,respectively, and a selector 482 used to select one from the standardimage data 481 a, 481 b, 481 c, and 481 d. For example, datacorresponding to the brightness distribution according to the distancebetween the light 403 and the subject illustrated in FIGS. 15A to 15Dmay be used as the standard image data 481 a, 481 b, 481 c, and 481 d.

An output of the distance sensor 470 is supplied to a control unit 430e. The control unit 430 e acquires the distance to the subject based onthe supplied output of the distance sensor 470, and controls theselector 482 according to the acquired distance such that the standardimage data according to the distance is selected among the standardimage data 481 a to 481 d. The standard image data selected by theselector 482 among the standard image data 481 a to 481 d is output tothe operation unit 447.

In the above example, the standard image data selected stepwise on thedistance is supplied to the operation unit 447, but the presentinvention is not limited to this example. For example, a method ofexpressing the characteristic indicating the standard image describedwith reference to FIG. 4 through a function including a distance valueindicating a distance from a subject as a parameter and applying thisfunction to the standard image output unit 480 is also considered. Thecontrol unit 430 e supplies the distance value based on the output ofthe distance sensor 470 to the standard image output unit 480 as aparameter. The standard image output unit 480 performs an operation ofthe function indicating the characteristic of the standard image usingthe parameter supplied from the control unit 430 e, and supplies anoperation result to the operation unit 447 as the standard image datacorresponding to the distance indicated by the distance value.

Further, the control unit 430 e may supply the distance value based onthe output of the distance sensor 470 to the standard image output unit480, and the standard image output unit 480 may interpolate each of thestandard image data 481 a to 481 d based on the distance value andoutput the standard image data corresponding to the distance value.

FIG. 17 is an exemplary flowchart illustrating an image correctionprocess according to the second embodiment. For example, when an appaccording to the second embodiment is activated according to the user'soperation on the operating unit 402 in step S130, in step S131, thecontrol unit 430 e turns on the light 403 to emit light. In step S132,the control unit 430 e detects the distance to the subject (the sheet330) based on the output of the distance sensor 470.

In step S133, the control unit 430 e is on standby for a capturinginstruction. For example, when the capturing instruction operation isdetermined to have been performed on the operating unit 402, the controlunit 430 e causes the process to proceed to step S134. On the otherhand, when no capturing instruction operation is determined to have beenperformed on the operating unit 402, the control unit 430 e causes theprocess to return to step S133.

In step S134, the control unit 430 e reads the captured image data ofone frame from the frame memory 442, causes the captured image data ofone frame to be stored in the frame memory 445, and acquires thecaptured image data of the processing target. In step S135, the controlunit 430 e controls the selector 482 according to the distance detectedin step S132 such that the standard image data corresponding to thedistance is selected among the standard image data 481 a to 481 d. Then,the control unit 430 e controls the operation unit 447 such that thestandard image data selected and output by the selector 482 is inputfrom the standard image output unit 480 to the operation unit 447.

In step S136, the operation unit 447 corrects the captured image datastored in the frame memory 445 in step S134 according to Formulas (1) to(3) using the standard image data input step S135. The operation unit447 acquires the corrected image data that is corrected by thecorrection process (step S137). The operation unit 447 outputs theacquired corrected image data to an image extracting unit (notillustrated).

Specific Exemplary Configuration of Information Processing Device

FIG. 18 is an exemplary configuration of an information processingdevice that can be applied to the above embodiments and the abovemodified examples. FIG. 18 illustrates an exemplary configurationcapable of implementing the information processing device 400 describedabove with reference to FIG. 2, but the configuration of FIG. 18 can bealso applied to the information processing devices 400 a to 400 eaccording to the modified examples of the first embodiment and thesecond embodiment.

In the information processing device 400 illustrated in FIG. 18, a CPU490, a ROM 491, a RAM 492, and a display control unit 493 are connectedto a bus. A storage 495, a data I/F 496, an input device 497, and acommunication unit 498 are also connected to the bus. For example, thestorage 495 is a non-volatile semiconductor memory such as a flashmemory serving as a storage medium capable of storing data in anon-volatile manner. The present invention is not limited to thisexample, and a hard disk drive may be used as the storage 495.

In the information processing device 400, a light 403, a distance sensor470, and a camera module 404 are further connected to the bus. The light403 and the distance sensor 470 are connected to the bus via aninterface (not illustrated). The present invention is not limited tothis example, and the light 403 and the distance sensor 470 may beconnected directly to the CPU 490 via an interface (not illustrated).

The CPU 490 controls an operation of the information processing device400 in general using the RAM 492 as a work memory according to a programstored in the ROM 491 and the storage 495. The display control unit 493converts a display control signal generated by the CPU 490 into a signalthat can be displayed by a display device 494, and outputs the convertedsignal.

The storage 495 stores a program executed by the CPU 490 and variouskinds of data. The data i/F 496 receives data from the outside. Forexample, an interface according to a Universal Serial Bus (USB),institute of Electrical and Electronics Engineers 1394 (IEEE1394), orthe like may be applied as the data i/F 496.

The input device 497 receives a user input, and outputs a certaincontrol signal. For example, when the user operates the input device 497in response to the display performed by the display device 494, aninstruction may be output to the information processing device 400.Further, preferably, the input device 497 is configured integrally withthe display device 494 as a touch panel that outputs a control signalaccording to the position at which the input device 497 is pushed andtransmits an image of the display device 494.

The communication unit 498 performs wireless or wired communication witha network using a certain protocol. The camera module 404 performs acapturing function, a zoom function, and the like according to aninstruction given by the user's operation on the input device 497. Thecaptured image data obtained by the photography performed by the cameramodule 404 is transferred to the CPU 490 via the bus.

The control unit 410, the correction processing unit 411, and thestandard image output unit 412 are implemented by an application program(app) operating on the CPU 490. The application program may be stored inthe ROM 491 or the storage 495 in advance and supplied. The presentinvention is not limited to this example, and the application program isconfigured to be stored on a computer connected to a networkcommunicable with the communication unit 498 and supplied by downloadingvia the network. Further, the application program may be configured tosupplied or distribute via a network.

The present invention is not limited to this example, and theapplication program may be recorded in a computer readable recordingmedium such as a Compact Disk (CD) or a Digital Versatile Disk (DVD) asa file of an installable format or an executable format and provided. Inthis case, for example, the application program is supplied to theinformation processing device 400 through an external drive device or anexternal computer connected to the data I/F 496.

For example, the application program has a module configurationincluding the above-described respective units (the control unit 410,the correction processing unit 411, and the standard image output unit412), and as actual hardware, for example, as the CPU 490 reads theapplication program from the storage 495, and executes the applicationprogram, the respective units are loaded onto a main storage device (forexample, the RAM 492), and the respective units are generated on themain storage device.

Third Embodiment

Next, a third embodiment will be described. The third embodiment is anexample in which the information processing device 400 according to thefirst embodiment is applied to a display system that displays image databased on an image drawn by the user as if a fish were swimming in avirtual water tank.

The third embodiment is not limited to the example in which theinformation processing device 400 according to the first embodiment isapplied to the display system, and any one of the information processingdevices 400 a to 400 d according to the first embodiment and themodified examples of the first embodiment and the information processingdevice 400 e according to the second embodiment may be applied.

FIG. 19 is a block diagram illustrating an exemplary configuration of adisplay system according to the third embodiment. Referring to FIG. 19,the display system 1 includes a personal computer (PC) 10, and one ormore projector devices (PJ) 11 ₁, 11 ₂, and 11 ₃. The PC 10 can performwireless or wired data communication with the information processingdevice 400.

Referring to FIG. 19, a user 23 activates, for example, the appaccording to the first embodiment in the information processing device400, and captures a sheet 21 on which a user image 22 is drawn. Forexample, the information processing device 400 turns on the light 403 toemit light when the app is activated according to the flowchart of FIG.3 (step S100 and step S101), and acquires captured image data andstandard image data according to the user 23's capturing instruction onthe information processing device 400 (step S102 to step S104). Then,the information processing device 400 corrects the captured image dataaccording to Formulas (1) to (3) using the standard image data, andacquires image data obtained by correcting the captured image data (stepS105 and step S106).

The information processing device 400 transmits the image data obtainedby correcting the captured image data to the PC 10. The PC 10 performscertain image processing on the image data transmitted from theinformation processing device 400, and supplies resultant data to theprojector devices (PJs) 11 ₁, 11 ₂, and 11 ₃ as display image data. Theprojector devices 11 ₁, 11 ₂, and 11 ₃ project images 13 ₁, 13 ₂, and 13₃ onto a screen 12 according to the display image data supplied from thePC 10.

Further, when the images 13 ₁, 13 ₂, and 13 ₃ are projected onto thesingle screen 12 through a plurality of projector devices (PJs) 11 ₁, 11₂, and 11 ₃ as illustrated in FIG. 19, an overlapping portion ispreferably formed in adjacent portions of the images 13 ₁, 13 ₂, and 13₃. In the example of FIG. 19, the images 13 ₁, 13 ₂, and 13 ₃ projectedonto the screen 12 are captured by a camera 14, and the PC 10 controlsthe images 13 ₁, 13 ₂, and 13 ₃ or the projector devices (PJs) 11 ₁, 11₂, and 11 ₃ based on the captured image data such that an image of theoverlapping portion is adjusted.

In this configuration, for example, the user 23 drawing the user image22 on the sheet 21 by hand, and the image of the sheet 21 is captured bythe information processing device 400. The information processing device400 corrects the captured image data using the standard image data asdescribed above, and transmits the corrected image data to the PC 10.The PC 10 decides pixels corresponding to the user image 22 from theimage data supplied from the information processing device 400, andextracts data of the region of the user image 22. At this time, in theimage data supplied from the information processing device 400, abrightness distribution of a portion other than a drawn portion becomesuniform through the correction process, and thus the region of the userimage 22 can be extracted. The PC 10 stores image data of the extractedregion of the user image 22 as the user image data.

On the other hand, the PC 10 generates an image data space having athree-dimensional (3D) coordinate system. Further, the PC 10 allocatescoordinates in the image data space to the user image data, and convertsthe user image data into data in the image data space. Hereinafter, theuser image data in the 3D image data space is referred to as a “userobject.” The PC 10 projects the 3D image data space including the userobject onto a two-dimensional (2D) image data plane, divides the imagedata generated by the projecting into the number of projector devices(PJs) 11 ₁, 11 ₂, and 11 ₃, and supplies the divided image data to eachof the projector devices (PJs) 11 ₁, 11 ₂, and 11 ₃.

Here, the PC 10 can give a motion in the image data space to the userobject. For example, the PC 10 obtains a feature quantity of the userimage data serving as a source of the user object, and generates eachparameter related to a motion including a deformation mode of the userobject based on the obtained feature quantity. The PC 10 applies theparameter to the user image data, and gives the motion in the image dataspace to the user object.

Thus, the user 23 can observe the user image 22 generated by his/herhand as an image moving in the 3D image data space. Further, the PC 10can cause a plurality of user objects to be included in the same imagedata space. Thus, the user 23 can observe, for example, each of aplurality of different user images 22 as an image moving in the 3D imagedata space by repeating the above operation.

FIG. 20 schematically illustrates an image data space having a 3Dcoordinate system generated by the PC 10 according to the thirdembodiment. In the third embodiment, an orthogonal coordinate systemhaving three coordinate axes (an x axis, a y axis, and a z axis)orthogonal to one another is used as the 3D coordinate system. In thefollowing description, as illustrated in FIG. 20, the x axis, the yaxis, and the z axis are assumed to be an axis in a height direction, anaxis in a width direction, and an axis in a depth direction, and the PC10 is assumed to generate an image data space 30 having a 3D coordinatesystem based on the axes of the height, the width, and the depth. In theexample of FIG. 20, user objects 40 ₁, 40 ₂, 40 ₃, 40 ₄, and 40 ₅ basedon the user image data are included in the image data space 30.

The PC 10 sets a definition region 31 serving as a space including aheight H, a width W, and a depth D of predetermined values to the imagedata space 30. A motion of each of the user objects 40 ₁, 40 ₂, . . .based on the user image data is restricted within the definition region31. In the third embodiment, a living thing living in the water such asa fish, a squid, an octopus, or a jellyfish can be assumed as the userimage 22 serving as a source of each of the user objects 40 ₁, 40 ₂, . .. , and the definition region 31 can be regarded as a virtual watertank. Hereinafter, the definition region 31 is referred to as a virtualwater tank 31 unless specially set forth.

FIG. 21 illustrates an exemplary configuration of the PC 10 according tothe third embodiment. In the PC 10 of FIG. 21, a CPU 110, a ROM 111, aRAM 112, and a display control unit 113 are connected to a bus 100. Inthe PC 10, a storage 114, a data I/F 115, and a communication I/F 116are further connected to the bus 100.

The CPU 110 controls the PC 10 in general according to a programpreviously stored in the ROM 111 and the storage 114 using the RAM 112as a work memory. The display control unit 113 is connected with amonitor 120, and converts a display control signal generated by the CPU110 into a signal that can be displayed by the monitor 120, and outputsthe converted signal. Further, the display control unit 113 can convertthe display control signal into a signal that can be displayed by theprojector devices 11 ₁, 11 ₂, and 11 ₃ and output the converted signal.

The storage 114 is a storage medium capable of storing data in anon-volatile manner, and, for example, a hard disk drive may be used.The present invention is not limited to this example, and a non-volatilesemiconductor memory such as a flash memory may be used as the storage114. The storage 114 stores a program executed by the CPU 110 andvarious kinds of data.

The data I/F 115 controls a data input and output with an externaldevice. For example, the data I/F 115 receives a signal from a pointingdevice such as a mouse or a keyboard (KBD) (not illustrated). Further,for example, the display control signal generated by the CPU 110 may beoutput from the data I/F 115 and supplied to the projector devices 11 ₁,11 ₂, and 11 ₃. An interface such as a USB or a Bluetooth (a registeredtrademark) may be applied as the data I/F 115.

The communication I/F 116 controls communication performed via a networksuch as the Internet or a Local Area Network (LAN). The PC 10 performscommunication with the information processing device 400 through thecommunication I/F 116.

FIG. 22 is an exemplary functional block diagram for describing afunction of the PC 10 according to the third embodiment. The PC 10includes an input unit 130, an image acquiring unit 131, a 3D spacegenerating unit 132, a region setting unit 133, a mode determining unit134, a parameter generating unit 135, an image control unit 136, and astorage unit 137.

The input unit 130 receives the image data including the user image 22drawn by hand by the user 23. For example, the input unit 130 includesthe function of the communication I/F 116, and receives the image datathat is obtained by capturing the sheet 21 including the user image 22drawn by hand by the user 23 and correcting using the standard image andtransmitted from the information processing device 400. Further, theinput unit 130 extracts the user image 22 from the input image data andacquires the extracted user image as the user image data.

The 3D space generating unit 132 generates the image data space 30 basedon 3D coordinate axes including three axes of the height, the width, andthe depth described above with reference to FIG. 20. For example, the 3Dspace generating unit 132 generates the image data space 30 as anaddress space on the RAM 112. The region setting unit 133 sets thedefinition region 31 (the virtual water tank 31) of the height H, thewidth W, and the depth D according to predetermined values to the imagedata space 30 generated by the 3D space generating unit 132.

The image acquiring unit 131 sets a certain point of view to the imagedata space 30, projects the image data space 30 onto the 2D image dataplane from the set point of view, and acquires image data to beprojected by the projector devices 11 ₁ to 11 ₃.

The mode determining unit 134 determines a deformation mode of the userobject (referred to as a “deformation mode”) that is allocated to theuser object when the user image data is included in the image data space30 based on the user image data acquired by the input unit 130. Theparameter generating unit 135 generates a parameter deciding theperformance related to the motion of the user object according to theuser image data based on the user image data acquired by the input unit130.

The image control unit 136 controls the motion of the user object in theimage data space 30 according to the deformation mode determined by themode determining unit 134 and the parameter generated by the parametergenerating unit 135. In other words, the image control unit 136allocates coordinates in the virtual water tank 31 in the image dataspace 30 to the user object, and performs control such that thecoordinates are continuously changed with the passage of time.

The storage unit 137 corresponds to the RAM 112, and stores, forexample, the user image data serving as a source of the user object. Thepresent invention is not limited to this example, and the storage 114may be used as the storage unit 137. For example, the mode determiningunit 134 and the parameter generating unit 135 determines thedeformation mode and generates the parameter using the user image datastored in the storage unit 137. Further, the image control unit 136 cancause the user image data to be included as the user object in the imagedata space 30 by allocating the coordinates in the virtual water tank 31to the user image data stored in the storage unit 137. Furthermore, theimage control unit 136 performs, for example, a deformation process or amovement process on the user object according to the deformation mode orthe parameter.

The input unit 130, the image acquiring unit 131, the 3D spacegenerating unit 132, the region setting unit 133, the mode determiningunit 134, the parameter generating unit 135, and the image control unit136 included in the PC 10 are implemented by a display control programthat is stored in, for example, the storage 114 and operates on the CPU110. The display control program is recorded in a computer readablerecording medium such as a CD, a Flexible Disk (FD), or a DVD as a fileof an installable format or an executable format and provided.

Further, the display control program executed by the PC 10 according toan embodiment may be configured to be stored on a computer connected toa network such as the Internet and supplied by downloading via thenetwork. Furthermore, the display control program executed by the PC 10according to an embodiment may be configured to supplied or distributevia a network such as the Internet. Moreover, the display controlprogram executed by the PC 10 according to an embodiment may beconfigured to be embedded in the ROM 111 or the like and provided.

The display control program executed by the PC 10 according to anembodiment has a module configuration including the above-describedrespective units (the input unit 130, the image acquiring unit 131, the3D space generating unit 132, the region setting unit 133, the modedetermining unit 134, the parameter generating unit 135, and the imagecontrol unit 136). As actual hardware, for example, as the CPU 110 readsthe display control program from a storage medium such as the storage114 or the ROM 111, and executes the display control program, therespective units are loaded onto a main storage device such as the RAM112, and the input unit 130, the image acquiring unit 131, the 3D spacegenerating unit 132, the region setting unit 133, the mode determiningunit 134, the parameter generating unit 135, and the image control unit136 are generated on the main storage device.

Next, a display control process according to an embodiment will bedescribed in further detail. FIG. 23 is an exemplary flowchartillustrating an overall flow of a display control process performed bythe display system according to the third embodiment. Before the processaccording to the flowchart is executed, the user 23 draws the user image22 on the sheet 21. Here, the user is assumed to draw the user image 22on a sheet having a predetermined format.

FIG. 24 illustrates an exemplary sheet for performing drawing by handaccording to the third embodiment. In a sheet 200 illustrated on theleft side of FIG. 24, a drawing region 210 in which a user image 212serving as a source of an object (the user object) desired to beincluded in the image data space 30 is drawn by hand and a title inputregion 211 in which a title of a picture drawn in the drawing region 210is input are arranged.

Further, markers 220 ₁, 220 ₂, and 220 ₃ are arranged on three of fourcorners of the sheet 200. By detecting the markers 220 ₁, 220 ₂, and 220₃ from the captured image obtained by capturing the image of the sheet200 through the information processing device 400, it is possible tounderstand the direction and size of the sheet 200. The markers 220 ₁,220 ₂, and 220 ₃ may be arranged on the corners of the drawing region210.

Referring to the flowchart of FIG. 23, the PC 10 receives the image datathat is obtained by capturing the image of the sheet 200 including theuser image 212 and then correcting using the standard image andtransmitted from the information processing device 400. In step S10, thereceived image data is input to the input unit 130.

In step S11, in the PC 10, the input unit 130 extracts the user imagedata from the input image data. First, the input unit 130 detects themarkers 220 ₁ to 220 ₃ from the image data. The input unit 130determines the direction and size of the image data based on thepositions of the detected markers 220 ₁ to 220 ₃ on the image data.

For example, the input unit 130 detects a corner on which none of themarkers 220 ₁ to 220 ₃ are positioned among the four corners of theimage data, and determines the direction of the image data based on theposition of the detected corner. In the example of FIG. 24, the corneron which none of the markers 220 ₁ to 220 ₃ are positioned is determinedto be a lower right corner of the image data. Based on this, thepositions of the markers 220 ₁ to 220 ₃ on the original image data canbe estimated. The input unit 130 measures the distance between themarkers 220 ₁ to 220 ₃, and compare the measured distance with a knowncorresponding distance previously stored in the storage 114 or the like.Distortion in the size of the original image data in the horizontal andvertical directions can be corrected Based on the comparison result.

Thus, the input unit 130 extracts the drawing region 210 and the titleinput region 211 from the image data based on the direction and size ofthe image data acquired as described above. An exemplary image based onthe image data of the drawing region 210 and the title input region 211extracted from the image data is illustrated on the right side of FIG.24. The input unit 130 further extracts a portion of the user image 212from the image data of the drawing region 210. For example, the inputunit 130 performs binarization on the image data by performing thresholddetermination on brightness of each of pixels of the image data of thedrawing region 210. The portion of the user image 212 is extracted onthe binarized image data. Image data of a minimum rectangular region 213that includes the portion of the user image 212 and has a direction ofthe bottom parallel to a direction of the bottom of the drawing region210 is used as the user image data. The input unit 130 causes thestorage unit 137 to store the user image data.

In step S12, in the PC 10, the mode determining unit 134 determines adeformation mode allocated to the user image data extracted in step S11.In an embodiment, the deformation mode is allocated to each user objectin the virtual water tank 31, and an operation of each user object inthe virtual water tank 31 is controlled based on the deformation mode.

In the third embodiment, three modes, that is, a first mode in which theuser object is vertically divided and deformed by moving a rear side ina moving direction, a second mode in which the user object is deformedby expanding and contracting the user object in the vertical direction,and a third mode in which the user object is horizontally divided anddeformed by moving a lower side are defined as the deformation mode. Inan embodiment, an aspect ratio R of the user image data is acquired, anyone of the first, second, and third modes that is allocated to the userobject based on the user image data is decided according to the acquiredaspect ratio R.

In step S13, in the PC 10, the parameter generating unit 135 generatesand decides a parameter p deciding the performance (hereinafter,referred to as “motion performance”) related to the motion of the userobject in the image data space 30 on the user object based on the userimage data. The motion of the user object in the image data space 30 iscontrolled according to the parameter p that is generated and decidedherein. For example, the parameter p includes a maximum speed andacceleration in the moving direction, a maximum value of angularacceleration in the horizontal and vertical directions, a maximum valueof a direction change speed in the vertical direction, and a maximumvalue (a maximum random number width) of a random number width in thehorizontal and vertical directions.

The parameter generating unit 135 obtains a feature quantity of the userimage 212, and decides each parameter p of the user object based on theuser image data based on the obtained feature quantity. The featurequantity of the user image data can be obtained based on a colordistribution or an edge quantity in the user image data. Further, thefeature quantity can be obtained based on bit information of pixelsconfiguring the user image data as described in Japanese Laid-openPatent Publication No. 2009-101122.

In step S14, the image control unit 136 sets a semi-transparent regionto the user image data. For example, the image control unit 136 detectsan inside region and an outside region of the portion of the user image212 extracted by the input unit 130 in the rectangular region 213including the user image data as described above with reference to FIG.24. The image control unit 136 sets pixels in the detected inside regionto be semi-transparent. Further, the image control unit 136 sets thedetected outside region to be transparent. The transparent andsemi-transparent setting of the image data can be implemented using aknown alpha blending technique.

By setting the semi-transparent region to the portion of the user image212 of the user image data as described above, the user object based onthe user image data in the virtual water tank 31 can look like a livingthing in the water.

Then, in step S15, the image control unit 136 sets initial coordinatesat which the user object is first displayed in the virtual water tank 31to the user object based on the user image data. In step S16, the imagecontrol unit 136 gives a motion to the user object to which the initialcoordinates are set in step S15, and causes the user object to start tomove. The image control unit 136 causes the user object to moveaccording to the parameter p while deforming the target user objectaccording to any one of the first to third deformation modes set to thetarget user object.

According to the present invention, an effect of making it possible toeasily extract a drawing region from a captured image obtained bycapturing an original is obtained.

The present invention can be implemented in any convenient form, forexample using dedicated hardware, or a mixture of dedicated hardware andsoftware. The present invention may be implemented as computer softwareimplemented by one or more network processing apparatus. The network cancomprise any conventional terrestrial or wireless communicationsnetwork, such as the Internet. The processing apparatus can compromiseany suitably programmed apparatuses such as a general purpose computer,personal digital assistant, mobile telephone (such as a WAP or3G-compliant phone) and so on. Since the present invention can beimplemented as software, each and every aspect of the present inventionthus encompasses computer software implemental on a programmable device.The computer software can be provided to the programmable device usingany storage medium for storing processor readable code such as a floppydisk, hard disk, CD ROM, magnetic tape device or solid state memorydevice.

The hardware platform includes any desired kind of hardware resourcesincluding, for example, a central processing unit (CPU), a random accessmemory (RAM), and a hard disk drive (HDD). The CPU may be implemented byany desired kind of any desired number of processor. The RAM may beimplemented by any desired kind of volatile or non-volatile memory. TheHDD may be implemented by any desired kind of non-volatile memorycapable of storing a large amount of data. The hardware resources mayadditionally include an input device, an output device, or a networkdevice, depending on the type of the apparatus. Alternatively, the HDDmay be provided outside of the apparatus as long as the HDD isaccessible. In this example, the CPU, such as a cache memory of the CPU,and the RAM may function as a physical memory or a primary memory of theapparatus, while the HDD may function as a secondary memory of theapparatus.

Although the invention has been described with respect to specificembodiments for a complete and clear disclosure, the appended claims arenot to be thus limited but are to be construed as embodying allmodifications and alternative constructions that may occur to oneskilled in the art that fairly fall within the basic teaching herein setforth.

What is claimed is:
 1. A terminal device, comprising: a capturing unitthat captures a subject; a light emitting unit that emits light; adisplay unit, a processor, and a storage unit that contain a programthat causes the processor to execute a process of acquiring a drawingtarget drawn by a user from a captured image of the subject includingthe drawing target and causing the drawing target to be displayed on ascreen of the display unit, the program causes the processor to execute;controlling the light emitting unit to emit the light when the capturingunit captures the subject according to the user's operation, acquiringthe captured image that is obtained by capturing the subject irradiatedwith the light emitted from the light emitting unit, performing imagecorrection corresponding to a brightness distribution of the lightemitted from the light emitting unit on the captured image, andextracting the drawing target from the captured image acquired.
 2. Theterminal device according to claim 1, wherein the program further causesthe processor to execute, outputting a standard image based on thebrightness distribution of the light at a predetermined distance from alight source, the standard image being used to decide a corrective valueused to perform the image correction on the captured image, controllingthe capturing unit and the light emitting unit such that the lightemitted from the light emitting unit is irradiated in a capturingdirection of the capturing unit, and correcting the captured imageacquired based on the corrective value decided using the standard image.3. The terminal device according to claim 1, wherein the program furthercauses the processor to execute, controlling the light emitting unit toemit the light in a standby state for a capturing operation performed bythe user in order to acquire the captured image.
 4. The terminal deviceaccording to claim 3, wherein the program further causes the processorto execute, recognizing a frame image set to the subject in advance fromthe captured image output from the capturing unit in a standby state fora capturing operation performed by the user, acquiring the capturedimage including the frame image according to the capturing operationperformed by the user when recognizing the frame image in the capturedimage.
 5. The terminal device according to claim 4, wherein the programfurther causes the processor to execute, outputting a display frameimage for supplying the frame image to the user when a capturing targetcaptured by the capturing unit is displayed on the display unit,determining whether or not the frame image recognized from the capturedimage is positioned within a predetermined range on a position of thedisplay frame image in a display region in the display unit, andacquiring the captured image when it is determined that the frame imageis positioned within the predetermined range on the position within thedisplay region.
 6. The terminal device according to claim 4, wherein theprogram further cause the processor to execute, determining whether ornot a shape of the recognized frame image satisfies a predeterminedshape condition, and acquiring the captured image when the shape of theframe image recognized is a predetermined shape.
 7. The terminal deviceaccording to claim 2, wherein the program further causes the processorto execute, controlling the storage unit to store a plurality ofstandard images according to a distance from the light source, andreading the standard image according to a distance between the subjectand the terminal device from the storage unit to output the standardimage.
 8. A method for acquiring drawing target performed by a terminaldevice that includes a capturing unit that captures a subject, a lightemitting unit that emits light, a display unit, a processor, and astorage unit that contain a program that causes the processor to executea process of acquiring a drawing target drawn by a user from a capturedimage of the subject including the drawing target and causing thedrawing target to be displayed on a screen of the display unit, themethod comprising; controlling the light emitting unit to emit the lightwhen the capturing unit captures the subject according to the user'soperation, acquiring the captured image that is obtained by capturingthe subject irradiated with the light emitted from the light emittingunit, performing image correction corresponding to a brightnessdistribution of the light emitted from the light emitting unit on thecaptured image, and extracting the drawing target from the capturedimage acquired.
 9. The method according to claim 8, further comprising;outputting a standard image based on the brightness distribution of thelight at a predetermined distance from a light source, the standardimage being used to decide a corrective value used to perform the imagecorrection on the captured image, controlling the capturing unit and thelight emitting unit such that the light emitted from the light emittingunit is irradiated in a capturing direction of the capturing unit, andcorrecting the captured image acquired based on the corrective valuedecided using the standard image.
 10. The method according to claim 8,further comprising; controlling the light emitting unit to emit thelight in a standby state for a capturing operation performed by the userin order to acquire the captured image.
 11. The method according toclaim 10, further comprising; recognizing a frame image set to thesubject in advance from the captured image output from the capturingunit in a standby state for a capturing operation performed by the user,acquiring the captured image including the frame image according to thecapturing operation performed by the user when recognizing the frameimage in the captured image.
 12. The method according to claim 11,further comprising; outputting a display frame image for supplying theframe image to the user when a capturing target captured by thecapturing unit is displayed on the display unit, determining whether ornot the frame image recognized from the captured image is positionedwithin a predetermined range on a position of the display frame image ina display region in the display unit, and acquiring the captured imagewhen it is determined that the frame image is positioned within thepredetermined range on the position within the display region.
 13. Themethod according to claim 11, further comprising; determining whether ornot a shape of the recognized frame image satisfies a predeterminedshape condition, and acquiring the captured image when the shape of theframe image recognized is a predetermined shape.
 14. A non-transitorycomputer-readable recording medium that contains a computer program thatis installed in a terminal device that includes a capturing unit thatcaptures a subject, a light emitting unit that emits light, a displayunit, a processor, and a storage unit that contain a program that causesthe processor to execute a process of acquiring a drawing target drawnby a user from a captured image of the subject including the drawingtarget and causing the drawing target to be displayed on a screen of thedisplay unit, the program causes the processor to execute; controllingthe light emitting unit to emit the light when the capturing unitcaptures the subject according to the user's operation, acquiring thecaptured image that is obtained by capturing the subject irradiated withthe light emitted from the light emitting unit, performing imagecorrection corresponding to a brightness distribution of the lightemitted from the light emitting unit on the captured image, andextracting the drawing target from the captured image acquired.
 15. Thenon-transitory computer-readable recording medium according to claim 14,wherein the program further causes the processor to execute, outputtinga standard image based on the brightness distribution of the light at apredetermined distance from a light source, the standard image beingused to decide a corrective value used to perform the image correctionon the captured image, controlling the capturing unit and the lightemitting unit such that the light emitted from the light emitting unitis irradiated in a capturing direction of the capturing unit, andcorrecting the captured image acquired based on the corrective valuedecided using the standard image.
 16. The non-transitorycomputer-readable recording medium according to claim 14, wherein theprogram further causes the processor to execute, controlling the lightemitting unit to emit the light in a standby state for a capturingoperation performed by the user in order to acquire the captured image.17. The non-transitory computer-readable recording medium according toclaim 16, wherein the program further causes the processor to execute,recognizing a frame image set to the subject in advance from thecaptured image output from the capturing unit in a standby state for acapturing operation performed by the user, acquiring the captured imageincluding the frame image according to the capturing operation performedby the user when recognizing the frame image in the captured image. 18.The non-transitory computer-readable recording medium according to claim17, wherein the program further causes the processor to execute,outputting a display frame image for supplying the frame image to theuser when a capturing target captured by the capturing unit is displayedon the display unit, determining whether or not the frame imagerecognized from the captured image is positioned within a predeterminedrange on a position of the display frame image in a display region inthe display unit, and acquiring the captured image when it is determinedthat the frame image is positioned within the predetermined range on theposition within the display region.
 19. The non-transitorycomputer-readable recording medium according to claim 17, wherein theprogram further cause the processor to execute, determining whether ornot a shape of the recognized frame image satisfies a predeterminedshape condition, and acquiring the captured image when the shape of theframe image recognized is a predetermined shape.
 20. The non-transitorycomputer-readable recording medium according to claim 15, wherein theprogram further causes the processor to execute, controlling the storageunit to store a plurality of standard images according to a distancefrom the light source, and reading the standard image according to adistance between the subject and the terminal device from the storageunit to output the standard image.